[https://nvbugs/6567403][fix] Waive hanging MoE multi-GPU tests on Hopper - #17344
Conversation
…pper Three test_moe_module.py multi-GPU cases hang on DGX H100 and are killed by the inner pytest timeout after ~2427s, pushing DGX_H100-4_GPUs-PyTorch-Others-1 past its Slurm wall limit. Waived on Hopper only; other platforms keep coverage. Signed-off-by: xxi <xxi@nvidia.com>
WalkthroughThe integration test waiver list now skips three configurable multi-GPU MoE tests on DGX_H100. Each skip entry references nvbugs/6567403. ChangesConfigurable multi-GPU MoE test waivers
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Around line 202-204: Add the repository-standard NVIDIA copyright header at
the beginning of tests/integration/test_lists/waives.txt, using 2026 as the
year, before the existing waiver entries.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 660873c6-53d2-49d3-bba1-186be3044b2c
📒 Files selected for processing (1)
tests/integration/test_lists/waives.txt
|
/bot run --stage-list "DGX_H100-4_GPUs-PyTorch-Others-1" |
|
PR_Github #64255 [ run ] triggered by Bot. Commit: |
|
/bot skip --comment "premerge hang" |
|
PR_Github #64260 [ skip ] triggered by Bot. Commit: |
There was a problem hiding this comment.
Format checks out. full:DGX_H100/ is the right token here: parse_test_name takes the test_prefix.split("-")[0] in s branch, and for stage DGX_H100-4_GPUs-PyTorch-Others-1 that first segment is DGX_H100, so all three entries resolve to exactly the node IDs l0_dgx_h100.yml:176,177,180 generate. full:H100/ would have silently missed this stage ("DGX_H100" is not a substring of H100/...) — that is the usual mis-scoping failure mode, and you avoided it.
No leakage to B200/B300/H200/GB200/GB300 or to single-GPU H100-* stages. Not blocking on it, but this trades away the only DGX-H100 EPLB case plus CUTLASS W8A16/W4A16_MXFP4 — a hang, not a flake — so 6567403 wants a real root-cause, not just the unblock.
|
PR_Github #64255 [ run ] completed with state |
|
PR_Github #64260 [ skip ] completed with state |
4 participants
Summary
Waive three MoE multi-GPU unit tests on Hopper only, tracked by NVBug 6567403.
On DGX H100 (SM90) these three cases hang instead of finishing in their usual ~1 minute. Each is killed by the inner pytest
--timeout=2400after ~2427s and reported asFAILED:unittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu -k "CUTLASS and W4A16_MXFP4"unittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu -k "CUTLASS and W8A16"unittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu_eplbThe accumulated hangs push
DGX_H100-4_GPUs-PyTorch-Others-1past its 240-minute Slurm wall limit, so the stage isscancel'd and never uploadsresults.xml.Observed signature (
DGX_H100-4_GPUs-PyTorch-Others-1):Scope of the waiver
The entries are scoped with
full:DGX_H100/, so only the Hopper stage is affected. B200, B300, H200 and GB200/GB300 keep full coverage.Resolution was verified against the repo's own
parse_test_list_lineswith the real--test-prefixvalues used by CI:--test-prefixDGX_H100-4_GPUs-PyTorch-Others-1DGX_H100-4_GPUs-PyTorch-Others-1/test_unittests.py::test_unittests_v2[...]DGX_B200-4_GPUs-PyTorch-Others-1DGX_H100/test_unittests.py::test_unittests_v2[...]DGX_H200-4_GPUs-PyTorch-Post-Merge-1DGX_H100/test_unittests.py::test_unittests_v2[...]GB300-4_GPUs-PyTorch-Others-1DGX_H100/test_unittests.py::test_unittests_v2[...]Test plan
N/A — test waiver only. No product code is touched.
Dev Engineer Review
DGX_H100waiver entries intests/integration/test_lists/waives.txt.nvbugs/6567403.DGX_H100.QA Engineer Review
tests/integration/test_lists/waives.txt.test_configurable_moe_multi_gpu -k "CUTLASS and W4A16_MXFP4"test_configurable_moe_multi_gpu -k "CUTLASS and W8A16"test_configurable_moe_multi_gpu_eplb